Maximum Entropy Approximation of Small Sample Distributions

In this paper, we propose a simple approach for approximating small sample distributions based on the method of Maximum Entropy (maxent) density. Unlike previous studies that use the maxent method as a curve-fitting tool, we focus on the “small sample asymptotics” of the approximation. We show that this method obtains the same asymptotic order of accuracy as that of the classical Edgeworth expansion and the exponential Edgeworth expansion by Field and Hampel (1982). In addition, the maxent approximation attains the minimum Kullback-Leibler distance among all approximations of the same exponential family. For symmetric fat-tailed distributions, we adopt the dampening approach in Wang (1992) to ensure the integrability of maxent densities and effectively extend the maxent method to accommodate all admissible skewness and kurtosis space. Our numerical experiments show that the proposed method compares competitively with and often outperforms the Edgeworth and exponential Edgeworth expansions, especially when the sample size is small.